<html>
<head>
  <title>数据分析操作</title>
  <basefont face="微软雅黑" size="2" />
  <meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
  <meta name="exporter-version" content="Evernote Windows/307027 (zh-CN, DDL); Windows/6.1.0 (Win32);"/>
  <style>
    body, td {
      font-family: 微软雅黑;
      font-size: 10pt;
    }
  </style>
</head>
<body>
<a name="518"/>
<h1>数据分析操作</h1>

<div>
<span><div><div><div style="text-align: center;"><font style="font-size: 24pt;"><span style="color: rgb(28, 51, 135); font-size: 24pt; font-weight: bold;">数据图表展示</span></font></div><div><img src="数据分析操作_files/v2-f39a4d5347f1e9b315b78f295249e979_r.jpg" type="image/jpeg" data-filename="v2-f39a4d5347f1e9b315b78f295249e979_r.jpg"/></div><div><font style="font-size: 18pt;"><span style="line-height: 1.45; color: rgb(28, 51, 135); font-size: 18pt; font-weight: bold;">点分布图：</span></font></div><div style="box-sizing: border-box; padding: 8px; font-size: 12px; border-top-left-radius: 4px; border-top-right-radius: 4px; border-bottom-right-radius: 4px; border-bottom-left-radius: 4px; background-color: rgb(251, 250, 248); border: 1px solid rgba(0, 0, 0, 0.14902);"><div><span style="font-size: 9pt; background-color: rgb(251, 250, 248); color: rgb(51, 51, 51); font-family: Monaco;">#scatter plot totalbsmtsf/saleprice</span></div><div><span style="font-size: 9pt; background-color: rgb(251, 250, 248); color: rgb(51, 51, 51); font-family: Monaco;">var = 'TotalBsmtSF'</span></div><div><span style="font-size: 9pt; background-color: rgb(251, 250, 248); color: rgb(51, 51, 51); font-family: Monaco;">data = pd.concat([df_train['SalePrice'], df_train[var]], axis=1)</span></div><div><span style="font-size: 9pt; background-color: rgb(251, 250, 248); color: rgb(51, 51, 51); font-family: Monaco;">data.plot.scatter(x=var, y='SalePrice', ylim=(0,800000));</span></div></div><div><img src="数据分析操作_files/Image.png" type="image/png" data-filename="Image.png"/></div><div><br/></div><div><font style="font-size: 18pt;"><span style="color: rgb(28, 51, 135); font-size: 18pt; font-weight: bold;">柱状分布图：</span></font></div><div style="box-sizing: border-box; padding: 8px; font-size: 12px; border-top-left-radius: 4px; border-top-right-radius: 4px; border-bottom-right-radius: 4px; border-bottom-left-radius: 4px; background-color: rgb(251, 250, 248); border: 1px solid rgba(0, 0, 0, 0.14902);"><div><span style="font-size: 9pt; background-color: rgb(251, 250, 248); color: rgb(51, 51, 51); font-family: Monaco;">sns.distplot(df_train['SalePrice'])</span></div></div><div><img src="数据分析操作_files/Image [1].png" type="image/png" data-filename="Image.png"/></div><div><br/></div><div><font style="font-size: 18pt;"><span style="color: rgb(28, 51, 135); font-size: 18pt; font-weight: bold;">烛台图表：</span></font></div><div style="box-sizing: border-box; padding: 8px; font-size: 12px; border-top-left-radius: 4px; border-top-right-radius: 4px; border-bottom-right-radius: 4px; border-bottom-left-radius: 4px; background-color: rgb(251, 250, 248); border: 1px solid rgba(0, 0, 0, 0.14902);"><div><span style="font-size: 9pt; background-color: rgb(251, 250, 248); color: rgb(51, 51, 51); font-family: Monaco;">#box plot overallqual/saleprice</span></div><div><span style="font-size: 9pt; background-color: rgb(251, 250, 248); color: rgb(51, 51, 51); font-family: Monaco;">var = 'OverallQual'</span></div><div><span style="font-size: 9pt; background-color: rgb(251, 250, 248); color: rgb(51, 51, 51); font-family: Monaco;">data = pd.concat([df_train['SalePrice'], df_train[var]], axis=1)</span></div><div><span style="font-size: 9pt; background-color: rgb(251, 250, 248); color: rgb(51, 51, 51); font-family: Monaco;">f, ax = plt.subplots(figsize=(8, 6))</span></div><div><span style="font-size: 9pt; background-color: rgb(251, 250, 248); color: rgb(51, 51, 51); font-family: Monaco;">fig = sns.boxplot(x=var, y=&quot;SalePrice&quot;, data=data)</span></div><div><span style="font-size: 9pt; background-color: rgb(251, 250, 248); color: rgb(51, 51, 51); font-family: Monaco;">fig.axis(ymin=0, ymax=800000);</span></div></div><div><img src="数据分析操作_files/Image [2].png" type="image/png" data-filename="Image.png"/></div><div><br/></div><div><br/></div><h4><span style="color: rgb(28, 51, 135); font-size: 18pt;">Correlation matrix (heatmap style)：</span></h4><div style="box-sizing: border-box; padding: 8px; font-size: 12px; border-top-left-radius: 4px; border-top-right-radius: 4px; border-bottom-right-radius: 4px; border-bottom-left-radius: 4px; background-color: rgb(251, 250, 248); border: 1px solid rgba(0, 0, 0, 0.14902);"><div><span style="font-size: 9pt; background-color: rgb(251, 250, 248); color: rgb(51, 51, 51); font-family: Monaco;">#correlation matrix</span></div><div><span style="font-size: 9pt; background-color: rgb(251, 250, 248); color: rgb(51, 51, 51); font-family: Monaco;">corrmat = df_train.corr()</span></div><div><span style="font-size: 9pt; background-color: rgb(251, 250, 248); color: rgb(51, 51, 51); font-family: Monaco;">f, ax = plt.subplots(figsize=(12, 9))</span></div><div><span style="font-size: 9pt; background-color: rgb(251, 250, 248); color: rgb(51, 51, 51); font-family: Monaco;">sns.heatmap(corrmat, vmax=.8, square=True);</span></div></div><div><img src="数据分析操作_files/Image [3].png" type="image/png" data-filename="Image.png"/></div><div style="box-sizing: border-box; padding: 8px; font-size: 12px; border-top-left-radius: 4px; border-top-right-radius: 4px; border-bottom-right-radius: 4px; border-bottom-left-radius: 4px; background-color: rgb(251, 250, 248); border: 1px solid rgba(0, 0, 0, 0.14902);"><div><span style="font-size: 9pt; background-color: rgb(251, 250, 248); color: rgb(51, 51, 51); font-family: Monaco;">#saleprice correlation matrix</span></div><div><span style="font-size: 9pt; background-color: rgb(251, 250, 248); color: rgb(51, 51, 51); font-family: Monaco;">k = 10 #number of variables for heatmap</span></div><div><span style="font-size: 9pt; background-color: rgb(251, 250, 248); color: rgb(51, 51, 51); font-family: Monaco;">cols = corrmat.nlargest(k, 'SalePrice')['SalePrice'].index</span></div><div><span style="font-size: 9pt; background-color: rgb(251, 250, 248); color: rgb(51, 51, 51); font-family: Monaco;">cm = np.corrcoef(df_train[cols].values.T)</span></div><div><span style="font-size: 9pt; background-color: rgb(251, 250, 248); color: rgb(51, 51, 51); font-family: Monaco;">sns.set(font_scale=1.25)</span></div><div><span style="font-size: 9pt; background-color: rgb(251, 250, 248); color: rgb(51, 51, 51); font-family: Monaco;">hm = sns.heatmap(cm, cbar=True, annot=True, square=True, fmt='.2f', annot_kws={'size': 10}, yticklabels=cols.values, xticklabels=cols.values)</span></div><div><span style="font-size: 9pt; background-color: rgb(251, 250, 248); color: rgb(51, 51, 51); font-family: Monaco;">plt.show()</span></div></div><div><img src="数据分析操作_files/Image [4].png" type="image/png" data-filename="Image.png"/></div><div><br/></div><div><br/></div><div><font color="#1C3387" style="font-size: 18pt;"><span style="color: rgb(28, 51, 135); font-size: 18pt; font-weight: bold;">pairplot：</span></font></div><div style="box-sizing: border-box; padding: 8px; font-size: 12px; border-top-left-radius: 4px; border-top-right-radius: 4px; border-bottom-right-radius: 4px; border-bottom-left-radius: 4px; background-color: rgb(251, 250, 248); border: 1px solid rgba(0, 0, 0, 0.14902);"><div><span style="font-size: 9pt; background-color: rgb(251, 250, 248); color: rgb(51, 51, 51); font-family: Monaco;">#scatterplot</span></div><div><span style="font-size: 9pt; background-color: rgb(251, 250, 248); color: rgb(51, 51, 51); font-family: Monaco;">sns.set()</span></div><div><span style="font-size: 9pt; background-color: rgb(251, 250, 248); color: rgb(51, 51, 51); font-family: Monaco;">cols = ['SalePrice', 'OverallQual', 'GrLivArea', 'GarageCars', 'TotalBsmtSF', 'FullBath', 'YearBuilt']</span></div><div><span style="font-size: 9pt; background-color: rgb(251, 250, 248); color: rgb(51, 51, 51); font-family: Monaco;">sns.pairplot(df_train[cols], size = 2.5)</span></div><div><span style="font-size: 9pt; background-color: rgb(251, 250, 248); color: rgb(51, 51, 51); font-family: Monaco;">plt.show();</span></div></div><div><img src="数据分析操作_files/Image [5].png" type="image/png" data-filename="Image.png"/></div><div><br/></div><div><br/></div><div style="text-align: center;"><font style="font-size: 24pt;"><span style="color: rgb(28, 51, 135); font-size: 24pt; font-weight: bold;">Missing data</span></font></div><ul><li><span style="font-size: 14pt;">How prevalent is the missing data?</span></li><li><span style="font-size: 14pt;">Is missing data random or does it have a pattern?</span></li></ul><div style="box-sizing: border-box; padding: 8px; font-size: 12px; border-top-left-radius: 4px; border-top-right-radius: 4px; border-bottom-right-radius: 4px; border-bottom-left-radius: 4px; background-color: rgb(251, 250, 248); border: 1px solid rgba(0, 0, 0, 0.14902);"><div><span style="font-size: 9pt; background-color: rgb(251, 250, 248); color: rgb(51, 51, 51); font-family: Monaco;">#missing data</span></div><div><span style="font-size: 9pt; background-color: rgb(251, 250, 248); color: rgb(51, 51, 51); font-family: Monaco;">total = df_train.isnull().sum().sort_values(ascending=False)</span></div><div><span style="font-size: 9pt; background-color: rgb(251, 250, 248); color: rgb(51, 51, 51); font-family: Monaco;">percent =(df_train.isnull().sum()/df_train.isnull().count()).sort_values(ascending=False)</span></div><div><span style="font-size: 9pt; background-color: rgb(251, 250, 248); color: rgb(51, 51, 51); font-family: Monaco;">missing_data = pd.concat([total, percent], axis=1, keys=['Total', 'Percent'])</span></div><div><span style="font-size: 9pt; background-color: rgb(251, 250, 248); color: rgb(51, 51, 51); font-family: Monaco;">missing_data.head(20)</span></div></div><div><div><br/></div><table style="border-collapse: collapse; min-width: 100%;"><colgroup><col style="width: 240px;"></col><col style="width: 287px;"></col></colgroup><tbody><tr><td style="border: 1px solid rgb(204, 204, 204); width: 240px; padding: 8px;"><div><img src="数据分析操作_files/Image [6].png" type="image/png" data-filename="Image.png"/></div></td><td style="border: 1px solid rgb(70, 146, 97); background-color: rgb(87, 183, 121); width: 287px; padding: 8px;"><div><ul><li><font color="#FFFFFF">We'll consider that when more than 15% of the data is missing, we should delete the corresponding variable and pretend it never existed.</font></li><li><font color="#FFFFFF">None of these variables seem to be very important, since most of them are not aspects in which we think about when buying a house (maybe that's the reason why data is missing?)</font></li><li><font color="#FFFFFF">'Garage<span style="font-style: italic;">X</span>' variables have the same number of missing data. I bet missing data refers to the same set of observations</font></li><li><font color="#FFFFFF">Since the most important information regarding garages is expressed by 'GarageCars' and considering that we are just talking about 5% of missing data, I'll delete the mentioned 'Garage<span style="font-style: italic;">X</span>' variables. The same logic applies to 'Bsmt<span style="font-style: italic;">X</span>' variables.</font></li><li><font color="#FFFFFF">Regarding 'MasVnrArea' and 'MasVnrType', we can consider that these variables are not essential. Furthermore, they have a strong correlation with 'YearBuilt' and 'OverallQual' which are already considered. Thus, we will not lose information if we delete 'MasVnrArea' and 'MasVnrType'.</font></li><li><span style="color: rgb(255, 255, 255);">Finally, we have one missing observation in 'Electrical'. Since it is just one observation, we'll delete this observation and keep the variable.</span></li></ul></div></td></tr></tbody></table></div><div><font style="font-size: 18pt; color: rgb(28, 51, 135);"><b>对于以上missing data 的处理</b></font></div><div style="box-sizing: border-box; padding: 8px; font-family: Monaco, Menlo, Consolas, &quot;Courier New&quot;, monospace; font-size: 12px; color: rgb(51, 51, 51); border-top-left-radius: 4px; border-top-right-radius: 4px; border-bottom-right-radius: 4px; border-bottom-left-radius: 4px; background-color: rgb(251, 250, 248); border: 1px solid rgba(0, 0, 0, 0.14902); background-position: initial initial; background-repeat: initial initial;-en-codeblock:true;"><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">#dealing with missing data</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">df_train = df_train.drop((missing_data[missing_data['Total'] &gt; 1]).index,1)</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">df_train = df_train.drop(df_train.loc[df_train['Electrical'].isnull()].index)</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">df_train.isnull().sum().max() #just checking that there's no missing data missing...</span></div></div><div><br/></div><div><br/></div><div><br/></div><div><br/></div><div style="text-align: center;"><font style="font-size: 24pt; color: rgb(28, 51, 135);"><b>Outliars</b></font></div><div style="text-align: center;"></div><h3 style="text-align: left;"><span style="font-weight: bold;"><font style="color: rgb(28, 51, 135); font-size: 18pt;">Univariate analysis</font></span></h3><div style="box-sizing: border-box; padding: 8px; font-family: Monaco, Menlo, Consolas, &quot;Courier New&quot;, monospace; font-size: 12px; color: rgb(51, 51, 51); border-top-left-radius: 4px; border-top-right-radius: 4px; border-bottom-right-radius: 4px; border-bottom-left-radius: 4px; background-color: rgb(251, 250, 248); border: 1px solid rgba(0, 0, 0, 0.14902); background-position: initial initial; background-repeat: initial initial;-en-codeblock:true;"><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">#standardizing data</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">saleprice_scaled = StandardScaler().fit_transform(df_train['SalePrice'][:,np.newaxis]);</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">low_range = saleprice_scaled[saleprice_scaled[:,0].argsort()][:10]</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">high_range= saleprice_scaled[saleprice_scaled[:,0].argsort()][-10:]</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">print('outer range (low) of the distribution:')</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">print(low_range)</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">print('\nouter range (high) of the distribution:')</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">print(high_range)</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);"><br/></span></div><div><br/></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">outer range (low) of the distribution:</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">[[-1.83820775]</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">[-1.83303414]</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">[-1.80044422]</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">[-1.78282123]</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">[-1.77400974]</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">[-1.62295562]</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">[-1.6166617 ]</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">[-1.58519209]</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">[-1.58519209]</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">[-1.57269236]]</span></div><div><br style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);"/></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">outer range (high) of the distribution:</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">[[ 3.82758058]</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">[ 4.0395221 ]</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">[ 4.49473628]</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">[ 4.70872962]</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">[ 4.728631 ]</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">[ 5.06034585]</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">[ 5.42191907]</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">[ 5.58987866]</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">[ 7.10041987]</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">[ 7.22629831]]</span></div></div><div><span style="-en-paragraph: true;">How 'SalePrice' looks with her new clothes:</span></div><ul><li>Low range values are similar and not too far from 0.</li><li>High range values are far from 0 and the 7.something values are really out of range.</li></ul><div><span style="-en-paragraph: true;">For now, we'll not consider any of these values as an outlier but we should be careful with those two 7.something values.</span></div><div><br/></div></div><div><br/></div><div><br/></div><h3><font style="font-size: 18pt; color: rgb(28, 51, 135);">Bivariate analysis</font></h3><div><img src="数据分析操作_files/Image [7].png" type="image/png" data-filename="Image.png" width="346"/></div><div><span style="-en-paragraph: true;">What has been revealed:</span></div><ul><li>The two values with bigger 'GrLivArea' seem strange and they are not following the crowd. We can speculate why this is happening. Maybe they refer to agricultural area and that could explain the low price. I'm not sure about this but I'm quite confident that these two points are <font style="color: rgb(227, 0, 0);"><b>not representative of the typical case</b></font>. Therefore, we'll define them as outliers and delete them.</li><li>The two observations in the top of the plot are those 7.something observations that we said we should be careful about. They look like two special cases, however they <font style="color: rgb(227, 0, 0);"><b>seem to be following the trend</b></font>. For that reason, we will keep them.</li></ul><div style="box-sizing: border-box; padding: 8px; font-family: Monaco, Menlo, Consolas, &quot;Courier New&quot;, monospace; font-size: 12px; color: rgb(51, 51, 51); border-top-left-radius: 4px; border-top-right-radius: 4px; border-bottom-right-radius: 4px; border-bottom-left-radius: 4px; background-color: rgb(251, 250, 248); border: 1px solid rgba(0, 0, 0, 0.14902); background-position: initial initial; background-repeat: initial initial;-en-codeblock:true;"><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">#deleting points</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">df_train.sort_values(by = 'GrLivArea', ascending = False)[:2]</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">df_train = df_train.drop(df_train[df_train['Id'] == 1299].index)</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">df_train = df_train.drop(df_train[df_train['Id'] == 524].index)</span></div></div><div><img src="数据分析操作_files/Image [8].png" type="image/png" data-filename="Image.png" width="356"/></div><div><br/></div><div><br/></div><div><br/></div><div style="text-align: center;"><font style="font-size: 24pt; color: rgb(28, 51, 135);"><b>Getting hard core</b></font></div><div><span style="-en-paragraph: true;">Who is 'SalePrice'?</span></div><div><span style="-en-paragraph: true;">The answer to this question lies in testing for the assumptions underlying the statistical bases for multivariate analysis. We already did some data cleaning and discovered a lot about 'SalePrice'. Now it's time to go deep and understand how 'SalePrice' complies with the statistical assumptions that enables us to apply multivariate techniques.</span></div><div><ul><li>Normality</li><li>Homoscedasticity</li><li>Linearity</li><li>Absence of correlated errors</li></ul></div><div><br/></div><div><br/></div><div><br/></div><div><br/></div><div><br/></div><div><br/></div><div><br/></div></div></span>
</div></body></html> 